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ABSTRACT 

During the past five years, staff of the American 
Institute for Research (AIR) in Palo Alto, California have been 
developing, field testing, and revising a guidance system especially 
suited to systems of individualized education. A primary aim of AIR'S 
Comprehensive Career Guidance System (CCGS) is to help each student 
plan wisely for the future. This paper has attempted to provide a 
rationale for, as well as examples of initial results from, the 
application of criterion- referenced measurement within an 
individualized career guidance system. Goals in the area of 'life' 
education have been identified and important indicator skills 
i solated. Educational experiences have been constructed which relate 
to these goals. Assessment of desired outcomes is attempted in terms 
of skill demonstration on items which relate directly to the goals of 
instruction. Results are used both to assist individual students in 
mastering desired skills and to improve the guidance system itself. 
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The overall aim of a school system should be to assist youth in selecting 
and achieving goals of personal significance. Students must be helped to recog- 
nize those skills that are significant and to schedule opportunities that will 
enable them not only to learn these in school situations but also to practice 
what they have learned in various relevant situations outside the school 
(Tyler, 1971). Within a total school system, no single operation should be 
more critical than the guidance function of assisting individual students with 
their career planning. The guidance portion of the school system should be 
predicated on the assumption that individual students need help both in setting 
their personal career goals (short- and long-term), and in identifying instruc- 
tional objectives in the different curriculum areas in order for them to attain 
their goals (Hamilton, 1971). 

In this context, the word "career" should not be limited only to the educa- 
tional and occupational choices of youth. Rather, it should be interpreted in 
a broader sense to include individualized planning for goals related to all 
areas of an individual's life (Katz, 1969). A comprehensive guidance system 
must be able to provide individual planning assistance to students to help 
them set career goals in such areas as their: (1) occupational future; (2) educa- 

tional future; (3) personal and social development; (4) development of learning 
skills; (5) development of community responsibility; and. (6) leisure-time acti- 
vities. 

During the past five years, staff of the American Institutes for Research 
in Palo Alto, California, have been developing, field testing, and revising a 
guidance system especially suited to systems of individualized education 
(Jones, et al , 1971). A primary aim of AIR's Comprehensive Career Guidance 
System (CCGS) is to help each student plan wisely for the future. A student 
is encouraged to realistically consider his chances of achieving a wide variety 
of potential goals, to select those goals which he is likely to reach and which 



are likely to afford him satisfying consequences, and to pursue his chosen goals 
with sufficient flexibility to take advantage of new opportunities and changing 
circumstances. 

The CCGS is being developed through the following activities: (1) identi- 

fication of youth development needs and the related behavioral objectives which 
state desired youth outcomes; (2) classification of objectives by commonalities 
and priorities which serve as guidelines for the design of guidance programs; 

(3) specification of alternative strategies which could be employed in potential 
guidance programs; (4) selection of strategies which seem most appropriate for 
certain groups of objectives and for groups of students with similar character- 
istics; (5) design , scheduling , and implementation of selected strategies by 
organizing instructional-counseling materials and procedures into individualized 
student learning units; (6) evaluation of the effectiveness of such units in 
helping students to achieve behavioral outcomes derived from the instructional 
objectives; and (7) revision of non-functioning elements based on evaluation 
results. 

The sixth activity noted above, evaluation, requires the development of 
tests and performance standards for measuring whether each student's own needs 
are being satisfied, i.e. , measuring the extent to which students are achieving 
both their short-term and long-range goals. This activity provides the feedback 
which is used to modify and correct the system's instructional-counseling elements. 

A major difficulty in evaluating the outcome of an individualized guidance 
system has been the unavailability of adequate criteria for "success" in indivi- 
dual planning, decision making, and self-management. Martin Katz (1969) has 
expressed the criteria that are particularly relevant to the outcomes of an 
individualized guidance system: 

Without directing the content of an individual 's choice, we do 
think we can help him in the process of choosing. This emphasis 
on process does not pretend to insure the "right" choice. Our 
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conviction is that in education enlightened processes are intrinsi- 
cally important. Therefore, we bend our efforts to increase the 
student's understanding of the factors involved in choice (imper- 
fect though our own understanding may be) so that he can take 
responsibility for his own decision making, examine himself and 
explore his options in a systematic and comprehensive way, take 
purposeful action in testing hypotheses about himself in various 
situations, and exercise flexibility in devising alternate plans. 

(P. 17) 

1 

The attainment of such skills is difficult to measure directly. In lieu of direct 
measurement, a set of tasks has been designed as behavioral indicators of a parti- 
cular process. Student performance on these tasks can then be assessed to deter- 
mine when instructional materials and procedures have succeeded in helping 
students acquire desired skills, when these materials and procedures have failed, 
and when they need further development to increase their effectiveness (Bandura, 

1969) . 

"Norm-referenced" measurement, the traditional strategy for assessing indi- 
vidual differences, is ill-suited to the evaluation requirements of an indivi- 
dualized guidance system because it provides a basis for comparing the perform- 
ance of two or more individuals with each other (Carver, 1970). Rather, evalua- 
tion instruments and procedures are needed which are capable of assessing each 
student's knowledge and attitudes relative to meaningful standards of individual 
achievement. "Criterion-referenced" measurement techniques are particularly 
appropriate for evaluating the effects of a guidance system designed to aid 
individual goal formulation and planning, since they are concerned primarily 
with measuring the degree of accomplishment of specified objectives (Nitko, 

1970) . 

Traditional methods of constructing tests, stressing median item difficulty 
and high correlation between each item and total test score, are inappropriate 
in the development of evaluation instruments for such "criterion-referenced" 



See Appendix A for a chart of the model of effective personal problem solving 
which was used as a guide in the specification of key student process skills 
for components of the CCGS. 



interpretation because those methods are intended to amplify test score varia- 
bility rather than decrease it. A "good" item in a CCGS test is one which will 
be answered correctly by 100% of those students who have mastered the objective 
upon which it is based and will be missed by 100% of those students who have 
not mastered the objective. Thus, such a test should not be faulted if, when 
administered after instruction, everyone obtains a perfect score (Popham and 
Husek, 1969). The process of developing test items based on a statement of' 
objectives is considerably simplified by using the method of comprehensive 
rationales (Flanagan, 1951). A comprehensive rationale consists of three parts. 
The first part is an illustration of the specific skill or activity, frequently 
using critical incidents describing especially effective or ineffective perform- 
ance. The second part of the rationale includes an analysis of the behavior 
described in part one. This analysis is particularly useful in demonstrating 
the importance of the behavior, permitting the avoidance of trivial items. The 
last part of the rationale proposes one or more specific test items which are 
felt to best represent effective performance on the defined activity. This 
also helps to insure that the behavior to be called for in the proposed item 
is within the immediate repertoire of the intended examinees. 

To define the activity necessitates a study of the requirements for success 
in that activity. A complete definition of what is meant by success is practi- 
cally identical with a statement of the procedure for obtaining criterion scores 
(Flanagan, 1962). A major assumption has thus been made that students must be 
able to demonstrate certain skills in personal problem solving related to goal 
selection and management rather than to reach conclusions about what goals and 
plans are best for them' as judged by independent "experts." 

The following tables present data which were collected from a sample of 
ninth and tenth grade students during a field test of CCGS components conducted 
in the San Jose Unified School District, San Jose, California. Evaluation 



instruments in which criterion items were tightly and strictly related to the 
outcomes of instruction with as high a degree of face validity as was possible 
were employed to assess student attainment of desired component outcomes. The 
particular outcomes focused on student acquisition of meaningful information 
and decision-processing skills in order to plan their educational and occupa- 
tional futures more wisely. 

Table 1 presents a summary of the responses of 30 students to 20 items 
related to the objective "to classify examples of the following factors you 
should consider when you think about your future educational and vocational 
goals: (a) abilities, (b) interests, (c) values, (d) physical traits, and 

(e) personal and social behavior." These items were written according to the 
techniques mentioned earlier and reflect the intended gain in "mastery" of a 
career guidance skill as a result of exposure to the CCGS program. The ,80% ,, 
accuracy level, which was chosen arbitrarily, was felt to represent an accept- 
able level of performance on this particular objective. It will be seen that 
the gain of 10% resulted in a posttest performance level of 25% mastery, far 
from the desired 100% level but realistic considering the brief exposure period. 

Table 2 presents the results of pre and posttests on another CCGS objective 
Again, this table shows rather substantial gains (40%) in "mastery level" of 
students exposed to the CCGS instructional experiences for 20 days; these gains 
are put in some additional perspective by the results shown for no-treatment 
students. Items in these tests should be examined to isolate those which seem 
to be answered correctly by both the pre-exposure treatment group and the 
no-treatment groups on pre and posttests. These items require revision to be 
more "discriminating" to the effects of instruction. 










TABLE 1 



PERCENT OF STUDENTS WHO ANSWERED 80% OF THE ITEMS 
RELATED TO THE FOLLOWING OBJECTIVE CORRECTLY 



To classify examples of the following factors you should consider when you think 
about your future educational and vocational goals: (a) abilities, (b) interests, 

(c) values, (d) physical traits, and (e) personal and social behavior. 




X 



0 



■X CCGS Program 

0 No Treatment 



Ss (N=20) 
Ss (N=10) 



TABLE 2 



PERCENT OF STUDENTS WHO ANSWERED 80% OF THE ITEMS 
RELATED TO THE FOLLOWING OBJECTIVE CORRECTLY 

To identify examples of the factors stated in the previous objective (abilities, 
interests, values, personal and social behavior, physical traits) after studying 
a description of a hypothetical student. 




X X CCGS Program Ss (N=20) 

0 o No Treatment Ss (N=10) 



Table 3 presents the results on an objective where the CCGS treatment was 
clearly ineffective in bringing about any gain in the performance of partici- 
pating students. Such data as these are useful in providing feedback for revi- 
sion of the instructional -counseling process. Either educational materials are 
not functioning in the intended manner or students did not receive sufficient 
exposure to these materials. Judgments about such matters of process are 
best made on the basis of other data. In this particular case, classroom 
observation and interviews with teachers suggested that few if any students 
had actually been adequately exposed to the materials for the objective in 
Table 3, due to unclear instructions. 

In summary, the present paper has attempted to provide a rationale for, 
as well as examples of initial results from, the application of criterion- 
referenced measurement within an individualized career guidance system. Goals 
in the area of "life" education have been identified and important indicator 
skills isolated. Educational experiences have been constructed which relate 
to these goals. Assessment of desired outcomes is attempted in terms of skill 
demonstration on items which relate directly to the goals of instruction. 
Results are used both to assist individual students in mastering desired 
skills and to improve the guidance system itself. 



TABLE 3 



PERCENT OF STUDENTS WHO ANSWERED 100% OF THE ITEMS 
RELATED TO THE FOLLOWING OBJECTIVE CORRECTLY 



To define the term "developed ability" and to describe three examples of scales 
on the Developed Abilities Performance Test. 



100 % 



75% 



X X 



Pretest Posttest 

X x CCGS Program [Attest ^20) 

° 0 No Treatment {ESS* ESI 



50% 



25% 
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APPENDIX A 





Personal Problem-Solving Model for the 
Comprehensive Career Guidance System 


Skill Area #1: 


Understanding the Problem 

Being willing to work toward setting and achieving goals in 
the various career areas (educational, occupational, personal- 
social, academic-learning, citizenship, leisure-time). 


Skill Area #2: 


Searching For and Using Information 

a. Personal information on abilities, interests, prefer- 
ences, etc., and 

b. Information about related opportunities in the various 
career areas. 


Skill Area #3: 


Locating Alternatives 

Thinking of several possible goals. 


Skill Area #4: 


Selecting Goals and Making Plans 

a. Choosing the goals (first and second choice) that seem 
like the best bet for the individual, and 

b. Making plans for reaching them. 


Skill Area #5: 


Carrying Out Plans 

Carrying out the plans, switching to the second best goal if 
necessary. 


Skill Area #6: 


Finding Out If It works 

a. Judging whether the first (or the second best) goal has 
been achieved, and 

b. Describing what helped and what hindered the individual's 
efforts along the way. 



